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1 .4.1 .1 Data Sources ° utsic 

The FBI maintains large databases of records. These records are recorded in both standard and 
vendor/developer unique formats. - ~ 
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Data processed by the system includes the following sources: 

• Approved case files from the FBI’s Automated Case Support (ACS) management system; 

• Electronic versions of the Joint Intelligence Committee Investigation (JICI) defined 
archived documents; 

• Secure Automated Messaging Network (SAMNet) message traffic; 

• IntelPlus (INTEL+) File Rooms; 


Note: Hardcopy versions of this document must be verified for correct version number with the IDW CM Document Control Library 

SAM NG-OM-005.1 1 ATO 
FOR OFFICIAL USE ONLY 
Page 2 


System Administrator Manual 
(SAM) 

Version 1.1 ATO 


IDW Operational Services 


SAM NG-OM-005.1 1 ATO 


Violent Gang and Terrorist Organization File (VGTOF) from the Cr imin al Justice 
Information Systems (CJIS) Division; 

Defense Advanced Research Projects Agency (DARPA) Translingual Information 
Detection, Extraction and Summarization (TIDES) Open Source Data. 
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The data flow begins at the external data sources. These sources include various FBI 
databases including ACS, SAMNET, Intelplus and several others.1 
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2.3.2 Special Projects Team (SPT) Subsystem 

The Special Projects Team (SPT) Subsystem allows for the rapid import of new specialized 
data sources. These data sources are not made available to the general IDW users but instead are 
provided to a small group of users who have a demonstrated “need-to-know”. The SPT System 
is similar in function to the IDW-S system. With the main difference is a different set of data 
sources. The SPT System allows it’s users to access not only the standard IDW Data Store but 
the specialized SPT Data Store. 

The table below lists the data and sources for the SPT Subsystem. 
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• Approved case files from the FBI’s Automated Case Support (ACS) Electronic 
Case File (ECF) case management system, 

• Secure Automated Messaging Network (SAMNet) message traffic, 


• IntelPlus File Rooms, 



• Violent Gang and Terrorist Organization File (VGTOF”) from the Criminal 
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• Approved case files from the FBI’s ACS ECF case management system; 

• Electronic versions of the JICI) defined archived documents; 

• SAMNet message traffic; 
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• Approved case files from the FBI’s Automated Case Support (ACS) Electronic 
Case File (ECF) case management system, 

• Secure Automated Messaging Network (SAMNet) message traffic, 

• IntelPlus File Rooms, 



Universal Name Index (UNI), and 

Violent Gang and Terrorist Organization File (VGTOF”) from the Criminal 
Justice Information Systems (CJIS) Division. 
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3.5. Data Processed 
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The IDW-DS system includes the following types of data. Representative samples of 
these data sets are used. Enough data is available in order to support development while 
minimizing the security risk. 

• Approved case files from the FBI’s ACS ECF case management system; 

• Electronic versions of the JICI) defined archived documents; 

• SAMNet message traffic; 
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Raw data is automatically transferred to the IDW-S system from approved data sources in 
accordance with an Intercon nection Agreement, Memorandum of Unders tanding, 
Memorandum of Agreement] |receives update 

data by the following means: 

• ^ ACS update data i] — — — 

• IntelPlus update da~ta\ 
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3.5.1. 1 IDW-S Core Subsystem Data Processed 
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Table 3. 5.1.1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 
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3.1.3.1 IDW-S Core Subsystem Data and Data Processing 

The IDW-S Core Subsystem holds and processes data from the following approved data sources: 

• Serialized Electronic Case File (ECF) documents in approved case files from the FBI’s 
Automated Case Support (ACS) system 

• IntelPlus Headquarters CT FileRooms 

• Secure Automated Messaging Network (SAMNet) message traffic 

• Electronic versions of the Joint Intelligence Committee Investigation (JICI) defined 
archived documents 

• Violent Gang and Terrorist Organization File (VGTOF) from the Criminal Justice 
Information Systems (CJIS) Division 

• Open Source News 
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3.9.9.1 IDW-S Core Subsystem Indirect Connections 
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Automated Transfer of Raw Data to IDW-S 
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Raw data is automatically transferred to the IDW-S system from approved data sources in 
accordance with an Interconnection Agreement, Memorandum of Understanding, 
Memorandum of Agreement or Interface Control Document. IDW-S VI receives update 
data by the following means: 

• ACS update data is transmitted to the IDW- S system] 

I 

• j IntelPlus update data is transmitted to the IDW-S system! 

• SAMNet update data is transmitted to the IDW-S system] 


• v GTDF data is transmitted to the ID W-S system^ 

• Open Source News data is transmitted to the IDW-S syster 
•' The JICI library is static and is not updated on IDW-S V 1 . 
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3.5.1 Data Processed 
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3.5.1.1 IDW-S Core Subsystem Data Processed 

Table 3. 5. 1.1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 
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Automated Transfer of Raw Data to IDW-S 


Outside the Scope 


Raw data is automatically transferred to the IDW-S system from approved data sources in 
accordance with an Interconnection Agreement, Memorandum of Understanding, Memorandum 
of Agreement or Interface Control Document. IDW-S VI receives update data by the following 


means: 


• ACS update data is transmitted to the IDW-S syster 


• IntelPlus update data is transmitted to the IDW-S system! 


SAMNet update data is transmitted to the IDW-S system! 


• VGTOF data is transmitted to the ID W-S system X 

• Open Source News data is transmitted to the EDW-S system\ 

• The JICI library is static and is not updated on IDW-S V 1 . 
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/data into individual serial records (in the case of ACS ECF), 

document s (IntelPlus, JICI, Open Source News), messages (SAMNetL or files 
(VGTOF)J 
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3.5.1 Data Processed I 

Outside the Scope 

1 


3.5.1.1 IDW-S Core Subsystem Data Processed 

Table 3.5. 1.1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 
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3.9.9 Indirect Connections 


3.9.9.1 IDW-S Core Subsystem Indirect Connections 


mm 




FBI Automated 
Case System 
(ACS) 

Secret 

FBI 

FTP 

FBI Intel Plus 

Secret 

FBI 

FTP 

FBI Secure 

Automated 

Messaging 

Network 

(SAMNet) 

Secret 

FBI 

FTP 

FBI National 
Crime 
Information 
Center (NCIC) 

Unclassified/SBU 

FBI 

CD-ROM 

FBI Document 

Conversion 

Laboratory 

(DocLab), 

Records 

Management 

Division (RMD), 

FBIHQ 

Secret 

FBI 

CD/DVD 
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INVESTIGATIVE DATA WAREHOUSE 

User Reference for 
IDW Version 1.1 
Applications 


IDW Training Office 
Federal Bureau of Investigation 
Department of Justice 
935 Pennsylvania Street • Room 10184 
Washington, D.C. 20535 
Phone 202.324.2606 



ACS* Automated Case Support (CT Case Files) Archived Investigative dating back through calendar year 
1995. Updates pushed (FTP) to IDW once per day. 


2» Intel Plus: Updates pushed (FTP) to IDW as needed (usually once per day). 



3. JKCI: Archived data from the Joint Intelligence Committee Investigation (JICI) regarding 9/11 as directed by 
the Congress of the United States. (Static file system) 

4* ORCON: Parsed data from SAMNet and ACS data collections with this caveat. Pushed to IDW once 
per day. 

5. SAMNet: SECRET and below cable traffic to/from FBI Headquarters. Pulled (FTP) from SAMNet 
server (IDW chronological job runs every 10 minutes; SAMNet posts new data 3 times per day). 

6. TFRG: Terrorist Financial records. Updates pushed (FTP) to IDW once per day. 

7. VGTOF: Violent Gang & Terrorist Offense File. Updated data file is provided in entirety by CD on a 
weekly basis. 

8. OS I NT: Open Source Research (OSINT): Current from the MiTAP server. 
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INVESTIGATIVE DATA WAREHOUSE 


SYSTEM SECURITY PLAN 


- SECRET 


INTRODUCTION 


The Federal Bureau of Investigation’s (FBI) Investigative Data Warehouse (IDW-S) is an 
initial data warehouse, content management and data mining system that will permit FBI 
investigative, analytical, administrative and intelligence personnel to access aggregated 
data previously only available through individual applications. The IDW-S system will be 
authorized to process classified national security data up to, and including, Secret. The 
IDW-S system is the successor of the Secure Counter-Terrorism/Collaboration Operational 
Prototype Environment (SCOPE”). 



Data processed by the system will include the following data sets: 0ut; 

• Approved case files from the FBI’s Automated Case Support (ACS) case 
management system; 

• Electronic versions of the Joint Intelligence Committee Investigation 

(JICI) defined archived documents; 


■Outside the Scope 
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• Secure Automated Messaging Network (SAMNet) message traffic; 

• IntelPlus File Rooms; and 

• Violent Gang and Terrorist Organization File (V GTOF”) from the 
C riminal Justice Information Systems (CJIS) Division; 


• Defense Advanced Research Projects Agency (DARPA) Translingual 
Information Detection, Extraction and Summarization (TIDES) Open 

SmiTY'p "Data 
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Data Sources 

IDW-S con tains data from the following data sources with security classification indicated 
in brackets: 

• Automated Case Support (ACS) System, Electronic Case File (ECF) 
Subsystem [Secret and below] 

• IntelPlus [Secret and below] 

• Secure Automated Message Network - Secret (SAMNet-S) [Secret and 
below] 

• NCIC Violent Gang and Terrorist Offender File (VGTOF) [Sensitive But 
Unclassified] 

• Joint Intelligence Committee Investigation (JICI) [Secret and below] 

• DARP A/TIDES Open Source News [Unclassified] 


Outside the Scope 



Data Ingest 

IDW-S receives data from source^ 

GTOF, Open Source News). 


](ACS ECF, IntelPlus, SAMNet) 




Received data ! [ nto individual serial records (in the case of ACS 

ECF), documents (IntelPlus, JICI, Open Source News), messages 
(SAMNet), or files (VGTOF). 



Outside the Scope 
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3.5. Data Processed 0utsi 

IDW-S processes the following data: 

3.5.1 ACS ECF 

IDW-S contains a subset of the ECF (Electronic Case File) subsystem of the 
Automated Case System (ACS). This subset consists of serials in those case 
classifications/subclassifications that have been officially sanctioned for inclusion 
in IDW. For each such serial, the ECF data includes metadata and text. IDW-S is 
synchronized against the ECF system once a day, a process which consists of 

receiving and pr ocessing the previous day’s increment o f ADD. MOD, and 

DELETE records! 


Outside the Scope 


3.5.2 InteLPlus 

IDW-S currently contains several IntelPlus counter-terrorism (CT) Filerooms: 
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3.5.3 SAMNet 

The Secure Automated Messaging N etwork ISAMNef) data source consists of ca ble 
traffic messages received by the FBlj 

\ SAMNet provides only ADD record 

types, and SAMNet data is updated in IDW-S three times a day. SAMNet datgj | 



3.5.4 VGTOF 


Violent Gang Terrorist Offender File (V GTOF) data is provided by the FBI 
National Crime Information Center (NCIC). VGTOF data includes two 
components: Data/metadata for each named individual/offender and potentially 
multiple JPEG images per individual/ 
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3.5.5 JICI 


The Joint Intelligence Committee Investigation (JICI) data collection was created following 
the September 11 attacks. Paper counter-terrorism files/folders from all FBI Field Offices 


tTICI is a static collection that 


is not updated] 


3.5.6 Open Source News 


The IDW-S V1.0 system contains an Open Source News library collected by the 
DARPA TIDES Program. These are primarily news source from around the world 
that are either in English or have been translated into English. Open Source News 
i'q re ceived in the form of text files (one per news article) 


JThe Open Source News data goes into IDW-S once a day. Open Source 
News data i; } | 

Open Source News material is derived from the following sources: 


Addis Ababa Tribune - http://www.addistribune.com/ 

Agencia Brasilia - http://www.radiobras.gov.br/ 

Al-Ahram (Egypt, weekly English version) - http://weekly.ahram.org.eg/ 
AllAfrica.com - http://allafrica.com/ 

Arabic News - http://www.arabicnews.com/ansub/ 

Asahi Shimbun - http://www.asahi.com/english/ 

Asia Times (Hong Kong) - http://www.atimes.com/ 

Bangkok Post - http://www.bangkokpost.net/ 

Christian Science Monitor - http://www.csmonitor.com/ 

Crescent International - http://www.muslimedia.com/mainpage.htm 
Daily Telegraph (London, England) - http://news.telegraph.co.uk/ 

Dawn (Karachi, Pakistan) - http://www.dawn.com/ 

Debka (Israel) - http://www.debka.com 

East Africa Daily Nation - www.nationaudio.com/News/DailyNation/Today/ 
Gulf News (UAE) - http://www.gulf-news.com/ 

Ha’aretz (Israel) - http://www.haaretzdaily.com/. 

IFRC Intemation Federation of the Red Cross - http://www.ifrc.org/ 

IRIN Integrated Regional Information Network - http://www.irinnews.org/ 
Iraq Press News Agency - http://www.iraqpress.org/ 
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Islamic Republic News Agency (Iran) - http://www.ima.com/en 
Jakarta Post - http://www.thejakartapost.com/ 

Janes - http://www.janes.com/ 

Jordan Times - http://www.jordantimes.com/ 

L'Osservatore Romano - www.vatican.va/news_services/or/or_eng/text.html 
Lagos (Nigeria) Guardian - http://www.guardiannewsngr.com/ 

Lahore (Pakistan) Nation - http://www.nation.com.pk/ 

Lebanon Daily Star (Beirut) - http://www.dailystar.com.lb/ 

Malaysian Star - http://thestar.com.my/ 

Manila Bulletin - http://www.mb.com.ph/ 

Manila Times - http://www.manilatimes.net/ 

Miami Herald - http://www.miami.com/ 

Moscow Times - http://www.themoscowtimes.com/ 

National Post and CP - http://www.canada.com/ 

New Straits Times (Kuala Lumpur) - http://www.emedia.com.my/Current_News/NST/ 
PETRA (Jordanian News Agency) - http://www.petra.gov.jo 
Pakistan Observer (Islamabad) - http://pakobserver.net/ 

Palestine Chronicle - http://palestinechronicle.com/ 

People's Daily (China) - http://english.peopledaily.com.cn/ 

Philippine Star - http://www.philstar.com/philstar/ 

Pravda - http://english.pravda.ru/ 

ProMed - epidemiology mailing list 

Russian Information Agency Novosti - http://en.rian.ru/ 

Russian Issues (Misc. Russian news) - http://www.therussianissues.com/ 

Russian Observer - http://www.russianobserver.com/ 

SABA (The News Agency of Yemen) - http://www.sabanews.gov.ye 
Saudi Gazette - http://www.saudigazette.com.sa/sgazette/ 

South African Dispatch - http://www.dispatch.co.za/ 

Sydney Morning Herald - http://www.smh.com.au 
Tehran Times - http://www.tehrantimes.com/ 

Times of India - http://timesofindia.indiatimes.com/cms.dll 
UNHCR UN High Commissioner on Refugees - http://www.unhcr.ch/ 

Ummah News - http://www.ummahnews.com/ 

Uzbekistan Report - http://www.uzreport.com/eng/ 

Washingtion Post - http://www.washingtonpost.com/ 

XinHua News Service - http://www.xinhuanet.com/english/ 

Yemen Times (weekly) - http://www.yementimes.com/ 

Yomiuri Shimbun (Japan) - http://www.yomiuri.co.jp/ 


3.5.7 Summary of Data Sources 





—i 
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ACS 

FBI Automated Case 
System (ACS) 

FTP 

IntelPlus 

FBI Intel Plus 

FTP 

SAMNet 

FBI Secure Automated 
Messaging Network 
(SAMNet) 

FTP 

VGTOF 

FBI National Crime 
Information Center (NCIC) 

CD-ROM 

JICI 

FBI Records Management 
Division (RMD), 
Document Laboratory 
(DocLab), FBIHQ 

Multiple 

Open 

Source 

News 

Translingual Information 
Detection, Extraction and 
Summarization (TIDES) 
Program, Defense Advancec 
Research Projects Agency 
(DARPA) 

CD-ROM 
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